
5 FIELD OF THE INVENTION 

The present invention relates to internetworking routers 
and, more particularly, to high-capacity routers having 
multiple interconnected chassis. 

10 

BACKGROUND OF THE INVENTION 

Ever-increasing usage of the Internet is expected to lead 
to even higher demands on the capacity of Internet 

15 routers than those which already persist today. However, 
the rate of growth of traffic through a given Web site or 
traffic point in the Internet may vary considerably 
amongst different Internet Service Providers (ISPs) . In 
some cases, the growth may be sudden and staggering, 

20 requiring huge increases in capacity on an almost 
instantaneous basis. In other cases, an anticipated 
increase in capacity might still be some time away, 
although a router with a basic switching capacity may 
need to be purchased immediately in order to satisfy an 

25 existing demand. 

A conventional approach to upgrading the routing capacity 
through a given traffic point is to simply replace the 
existing router with a new, higher-capacity device. The 
30 old router is either decommissioned or relegated to a 
less traffic-intensive area of the ISP's internal 
network. Unfortunately, this approach requires a 
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significant capital expenditure on the part of the ISP, 
since higher-capacity routers tend to be 

disproportionately more expensive than lower-capacity 
routers. Moreover, a capital expenditure of this nature 
5 is necessitated each time a capacity increase is required 
or desired. Additional disadvantages include the "down 
time" associated with installation of a new router, 
testing of new connections, changing suppliers and so on. 



10 Clearly, it would be advantageous to provide a scalable 
solution to the problem of accommodating traffic growth 
through a router. However, as now described, 

conventional router design makes this a near impossible 
feat. Specifically, a scalable router typically has two 

15 or more chassis, each of which contains multiple switch 
cards and line cards. The line cards have ports for 
interfacing with an external network. Internally to the 
router, the switch cards are connected to the line cards 
and to one another by a backplane on each chassis, and by 

20 direct interconnections across multiple chassis. 

In order to enhance the available switching capacity of 
the router, it may appear plausible to add one or more 
extra chassis but it should also be apparent that these 

25 additional chassis must somehow be connected to the 
existing chassis. As a result, existing hardware 

connections, both within and between he existing chassis, 
must be disconnected and then re-connected according to a 
different inter-chassis topology and a different intra- 

30 chassis interconnect pattern. Thus, while avoiding part 
of the capital expenditure associated with an outright 
replacement of the existing router, the conventional 
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solution has the disadvantage of requiring added 
installation and testing efforts, both of which are 
labour-intensive and prone to error. 

5 Hence, there remains a strong need in the industry to 
provide a scalable router that would be designed to 
accommodate changes in capacity without requiring 
replacement or disconnection of the existing inter- 
chassis or intra-chassis connection hardware. 

10 

SUMMARY OF THE INVENTION 

The present invention endeavours to obviate or mitigate 
one or more disadvantages of the prior art and may be 

15 summarized according to a first broad aspect as a router 
that includes a plurality of chassis, each chassis 
including a plurality of processing modules and a 
programmable interconnection module. Data connections 
are provided between each processing module on each 

20 chassis and the interconnection module on that chassis, 
and a data connection is provided between the 
interconnection module on each chassis and the 
interconnection module on at least one other chassis. 

25 In one specific embodiment, the data connections between 
the processing modules on each chassis and the 
interconnection module on that chassis are electrical and 
the data connections between the interconnection modules 
on different chassis are optical. 

30 

In another specific embodiment, the interconnection 
module on each chassis includes a plurality of electrical 
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input ports, a plurality of electrical output ports and a 
programmable switch fabric disposed therebetween, for 
providing selective connections between individual ones 
of the electrical input ports and corresponding ones of 
5 the electrical output ports. Additionally, each 

processing module on each chassis includes a plurality of 
electrical input ports, a plurality of electrical output 
ports and a processing fabric disposed therebetween. 
Moreover, the plurality of electrical input ports of each 

10 processing module on each chassis is connected to a 
respective subset of the electrical output ports of the 
interconnection module on that chassis, while the 
plurality of electrical output ports of each processing 
module on each chassis is connected to a respective 

15 subset of the electrical input ports of the 
interconnection module on that chassis 

In this way, the inter-chassis and intra-chassis 
connections pass through the programmable interconnection 

20 module, which may be used for concentrating the location 
of resources, such as opto-electronic and electro-optical 
converters, in a single card or unit. Moreover, the 
programmable nature of the switch fabric in each 
interconnection module is amenable to reconfiguration so 

25 as to support a change in that chassis' interconnection 
pattern that may be required for expanding the router's 
capacity. 

According to a second broad aspect, the present invention 
30 may be summarized as a chassis for use in building a 
scalable router. The chassis includes a plurality of 
processing modules, each processing module including a 



Page 4 




86177-23 

plurality of electrical input ports, a plurality of 
electrical output ports and a processing fabric disposed 
therebetween. The chassis also includes a programmable 
interconnection module, including a plurality of 
5 electrical input ports, a plurality of electrical output 
ports and a programmable switch fabric disposed 
therebetween, for selectively establishing connections 
between individual ones of the electrical input ports and 
corresponding ones of the electrical output ports in 
10 accordance with a connection map. 

A data connection is established between each processing 
module and the interconnection module, whereby a subset 
of the plurality of electrical input ports of each 

15 processing module on each chassis is connected to a 
respective subset of the electrical output ports of the 
interconnection module on that chassis and whereby a 
subset of the plurality of electrical output ports of 
each processing module on each chassis is connected to a 

20 respective subset of the electrical input ports of the 
interconnection module on that chassis. 

The chassis additionally includes a plurality of optical 
input ports and a plurality of optical output ports, for 

25 external connection to one or more other chassis of the 
router. The chassis also has a plurality of optical-to- 
electrical conversion units, each optical-to-electrical 
conversion unit being connected between a respective one 
of the optical input ports and a respective subset of the 

30 electrical input ports of the interconnection module. 
Finally, the chassis includes a plurality of electrical- 
to-optical conversion units, each electrical-to-optical 
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conversion unit being connected between a respective 
subset of the electrical output ports of the 
interconnection module and a respective one of the 
optical output ports. 

5 

According to a third broad aspect, the present invention 
may be summarized as a method of upgrading a router 
including a plurality of original chassis, each original 
chassis having a plurality of processing modules and a 

10 programmable interconnection module, wherein a data 
connection exists between each processing module on each 
original chassis and the interconnection module on the 
same original chassis and wherein a data connection 
exists between the interconnection module on each 

15 original chassis and the interconnection module on at 
least one other original chassis. The method includes 
providing at least one additional chassis, each 
additional chassis comprising a plurality of processing 
modules and a programmable interconnection module, 

20 wherein a data connection exists between each processing 
module on each additional chassis and the interconnection 
module on the same additional chassis. The method then 
includes establishing a data connection between the 
interconnection module on each additional chassis and the 

25 interconnection module on at least one original chassis, 
and establishing a data connection between the 
interconnection module on each additional chassis and the 
interconnection module on at least one other additional 
chassis. Finally/ the interconnection modules of the 

30 various original chassis are re-programmed. The 
interconnection modules of the various additional chassis 
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can be programmed prior to, or after, their connection to 
the original chassis. 

These and other aspects and features of the present 
invention will now become apparent to those of ordinary 
skill in the art upon review of the following description 
of specific embodiments of the invention in conjunction 
with the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

In the accompanying drawings: 

Fig. 1 depicts an interconnect pattern existing within a 
chassis in accordance with an embodiment of the present 
invention; 

Fig. 2 illustrates an example of the internal structure 
of an interconnection module in the chassis of Fig. 1 ; 

Fig. 3A shows a possible inter-chassis topology for 
building a router from two chassis; 

Fig. 3B shows a possible intra-chassis interconnection 
pattern for each of the chassis of Fig. 3A; 

Fig. 4A shows a possible inter-chassis topology for 
building a router from three chassis; 

Fig. 4B shows a possible intra-chassis interconnection 
pattern for each of the chassis of Fig. 4A; 
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Fig. 5A shows a possible inter-chassis topology for 
building a router from six chassis; 

Fig. 5B shows a possible intra-chassis interconnection 
5 pattern for each of the chassis of Fig. 5A; and 

Fig. 6 illustrates, in schematic form, a router made up 
of a set of "clusters" of chassis. 

10 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

With reference to Fig. 1, there is shown a chassis 
(shelf) 100, and more specifically a data interconnect 
pattern within the chassis 100, according to an 

15 embodiment of the present invention. Two or more chassis 
100 of the type shown in Fig. 1 may be used to build a 
router, in a manner to be described later on with 
reference to Figs. 3A through 5B. The chassis 100 
includes a plurality, of line cards 110 for interfacing 

20 with an external network 120, a plurality of processing 
cards 130 for providing processing and switching power, 
and an interconnection module 140. 

The line cards 110 provide an optical or electrical 
25 interface to the network 120. Data in the form of 
packets (or ATM cells or SONET frames or the like) are 
exchanged with the network 120 via the line cards 110. 
In a specific non-limiting example embodiment, the line 
cards 110 may be implemented as disclosed in U.S. patent 
30 application 09/870,766 to Norman et al . , filed on June 1, 
2001 and hereby incorporated by reference herein. In the 
embodiment illustrated in Fig. 1, the line cards 110 are 
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equipped with bi-directional functionality. In other 
embodiments, some of the line cards HQ may be input. line 
cards (for receiving packets from the network 120) , while 
other ones of the line cards 110 may be output line cards 
5 (for transmitting packets to the network 120) . 

The total number of line cards 110 in the illustrated 
embodiment is sixteen, each of which may be inserted into 
a corresponding slot (not shown) of the chassis 100. The 

10 sixteen line cards 110 are grouped into two sets denoted 
more precisely as 110 0 ,o-110 0 ,7 (set 0) and 110i f0 -H0i f7 
(set 1) . Each of the line cards 110 has a set of ports 
used for establishing one or more full-duplex electrical 
paths with a corresponding one of the processing cards 

15 130. By way of example, each of the line cards 110 may 
have four ports for establishing two full-duplex 2.5 Gbps 
(in each direction) electrical paths with a corresponding 
one of the processing cards 130. It should be 

understood, however, that the present invention is not 

20 limited to a specific number of line cards 110 or to a 
specific number of ports or paths per line card or to a 
specific path bandwidth. 

The total number of processing cards 130 in the chassis 
25 100 of the illustrated embodiment is four, each of which 
may be inserted into a corresponding slot (not shown) of 
the chassis 100. The four processing cards 130 include a 
set of two first-stage processing cards (denoted 130 0 ,o 
and 130 0 ,i) and a set of two second-stage processing cards 
30 (denoted 130i f0 and 130i,i) . Each of the first-stage 
processing cards includes a set of line-side ports for 
establishing sixteen full-duplex electrical paths L with 
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a corresponding set of the line cards 110. Each of the 
first- and second-stage processing cards 130 includes a 
set of switch-side ports for establishing either sixteen 
(first-stage) or thirty-two (second-stage) full-duplex 
5 electrical paths P with the interconnection module 140. 

As shown in Fig. 1, the line-side ports of each of the 
first- and second-stage processing cards 130 include a 
set of line-side input ports 132 and a set of line-side 

10 output ports 134. Similarly, the switch-side ports of 
each of the first- and second-stage processing cards 130 
include a set of switch-side input ports 136 and a set of 
switch-side output ports 138. Additionally, each of the 
first- and second-stage processing cards 130 has a a 

15 processing fabric 135, such as a switch fabric, connected 
between the input ports 132, 136 and the output ports 
134, 138. In a specific non-limiting example embodiment, 
the processing cards 130 may be implemented as disclosed 
in U.S. patent application 09/870,766 to Norman et al., 

20 filed on June 1, 2001 and hereby incorporated by 
reference herein. 

The paths L joining the line cards 110 and the processing 
cards 130 may be established through the use of a 

25 backplane or midplane configuration. In the illustrated 
embodiment, these paths are denoted L 0 -L 3 i, where, for the 
line cards 110 0 ,o to 110 0 ,7 in set 0, paths L 0 -L x are 
established between line card 110 0 ,o and processing card 
130 0 ,o/ paths L2-L3 are established between line card 110 0 ,i 

30 and processing card 130 0 ,o, paths L 4 -L 5 are established 
between line card 110 0 ,2 and processing card 130 0 ,o/ paths 
L 6 _L 7 ar e established between line card 110 0 ,3 and 
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processing card 130 0(0 , paths L 8 -L 9 are established between 
line card 110 0 ,4 and processing card 130 0 ,o, paths Li 0 -Ln 
are established between line card 110 0 ,5 and processing 
card 130 0 ,o, paths L 12 -Li 3 are established between line card 
5 110 0 ,6 and processing card 130 0 ,o and paths L14-L15 are 
established between line card 110 0 , 7 and processing card 
130 0 ,o- A similar interconnect pattern joins the line 
cards 110i /0 to 110i, 7 in set 1 to processing card 130 0 ,i- 

10 The paths P joining the processing cards 130 to the 
interconnection module 140 may also be established 
through the use of a backplane or midplane configuration. 
In the illustrated embodiment, there are ninety-six such 
paths and these are denoted P0-P95, where paths P0-P31 are 

15 established between processing card 130i, 0 and the 
interconnection module 140, paths P32-P63 are. established 
between processing card 130i #1 and the interconnection 
module 140, paths P 6 4~P79 are established between 
processing card 130 0 ,o and the interconnection module 140 

20 and paths Pso-Pss are established between processing card 
130 0 ,i and the interconnection module 14 0. 

Due to the fact that all first- and second-stage 
processing cards 130 are connected to the interconnection 

25 module 140, data arriving from the network 120 can be 
processed by one or several processing cards 130 in 
sequence. For example, data can be routed back into the 
network 120 after reaching one of the first-stage 
processing cards 130 0 ,o, 130 0 ,i or it can be forwarded to 

30 one of the second-stage processing cards 130i, 0 , 13.0 1(1 via 
the interconnection module 140. In another embodiment of 
the invention, the entire set of line cards 110 could be 
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connected to the entire set of processing cards 130, 
obviating a need for the distinction between "first- 
stage" and "second-stage" processing cards. Although the 
processing cards 130 are shown in Fig. 1 to be mutually 

5 non-interconnected, those skilled in the art should 
appreciate that it is within the scope of the present 
invention to provide additional data paths between pairs 
of processing cards 130. This may be advantageous when 
implementing a hypercube-based interconnect pattern, for 

10 example . 

It should be understood that the present invention is not 
limited to any particular number of processing cards 130, 
nor to any particular number of stages of processing 

15 cards 130, nor to any particular number of line-side 
ports or switch-side ports per processing card 130, nor 
to any particular interconnection pattern between the 
processing cards 130 and the interconnection module 140, 
nor to any particular implementation for achieving a 

20 connection between the processing cards 130 and either 
the line cards 110 or the interconnection module 14 0. 

The interconnection module 140 may be a separate card in 
the chassis 100 and includes a plurality of switch-side 

25 ports that establish paths P0-P95 with the switch-side 
ports of the processing cards 130. The interconnection 
module 140 further includes a plurality of optical ports 
A, B, C, D for establishing a plurality of full-duplex 
optical paths with the external world (i.e., with optical 

30 ports of other chassis of the router, to be described 
later on with reference to Figs. 3A to 5B) . In the 
illustrated embodiment, optical port A establishes 32 
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optical paths denoted A 0 -A 3 i, optical port B establishes 
24 optical paths denoted B 0 -B 23 , optical port C 
establishes 24 optical paths denoted C 0 -C 2 3/ and optical 
port D establishes 32 optical paths denoted D0-D31. 
However, it is to be understood that the present 
invention limits neither the number of optical ports nor 
the number of paths per optical port. 

The internal structure and functionality of the 
interconnection module 14 0 are now described with 
reference to Fig. 2. Paths Po-Pgs, which join the 
interconnection module 140 to the processing cards 130, 
are connected to electrical interfaces of a programmable 
switch fabric 200. More specifically, since each of the 
ninety-six paths P0-P95 is full-duplex, they are connected 
to ninety-six input electrical interfaces (denoted IN 0 - 
IN95) and ninety-six output electrical interfaces (denoted 
OUT0-OUT95) of the switch fabric 200. 

20 In addition, a set of 112 additional input electrical 
interfaces (denoted IN96-IN207) lead from the optical ports 
A, B, C, D via a bank of opto-electronic receivers 210, 
while a set of 112 additional output electrical 
interfaces (denoted OUT 96 -OUT 2 o7) lead to the optical ports 

25 A, B, C, D via a bank of electro-optical transmitters 
220. More specifically, electrical interfaces IN 96 -INi 2 7 
and OUT 9 6-OUTi 2 7 are associated with optical paths A 0 -A 3 i, 
electrical interfaces IN 128 -IN 151 and OUTi 28 -OUTi 5 i are 
associated with optical paths B 0 -B 23 , electrical 

30 interfaces IN 152 -INi 7 5 and OUT 152 -OUT 17 5 are associated with 
optical paths C 0 -C 23 , and electrical interfaces INi 76 -IN 20 7 
and OUTi7 6 -OUT 20 7 are associated with optical paths D 0 -D 3 i- 



10 
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Thus, by virtue of its input electrical interfaces IN 0 - 
IN207/ the switch fabric 200 can be on the receiving end 
of up to a total of 96 electrical data signals from the 

5 processing cards 130 and a further 112 electrical data 
signals from other chassis (via the optical ports A, B, 
C, D) , for a total of 208 received electrical signals. 
Similarly, by virtue of its output electrical interfaces 
OUT 0 -OUT 2 o7, the switch fabric 200 can transmit up to a 

10 total of 96 electrical data signals to the processing 
cards 130 and a further 112 electrical data paths to 
other chassis (via the optical ports A, B, C, D) , for a 
total of 208 transmitted electrical signals. 

15 In order to accommodate the switching requirement of the 
switch fabric 200, the latter may be implemented as a 
single, non-blocking cross-point switch matrix of the 
requisite size (208 x 208) . This would allow any of the 
electrical paths P0-P95 to be connected to any of the 

20 optical paths A 0 -A 3 i, B 0 -B 2 3, C 0 -C 2 3, and D 0 -D 3 i. However, 
it is noted that not all of the electrical signals 
received from the opto-electronic receivers 210 will need 
to be relayed to the processing cards 130. Rather, some 
of these may need to be immediately re-routed back to the 

25 optical ports via the electro-optical transmitters 220. 
Similarly, some of the electrical signals received from 
the processing cards 130 will need to be immediately re- 
routed back to the processing cards 130, although 
possibly to a different processing card than the one it 

30 originated from. Both of these types of immediate re- 
routing functionality can be referred to as "loopback" 
functionality. 
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As a result of this requirement for some degree ■ of 
loopback functionality, the switch fabric 200 can be 
constructed from two or more interconnected cross-point 
5 switches of smaller dimensionality, as illustrated in 
Fig. 2. Specifically, switch 230 takes care of switching 
a subset of 96 electrical signals (from the processing 
cards 130 to the external world), while switch 240 takes 
care of switching a subset of 112 electrical signals 
10 (from the external world to the processing cards 130) . 
Loopback functionality is made possible through an 
interconnection of a relatively small number of 
connections, in this case twenty-eight (28), between the 
two cross-point switches 230, 240. 

15 

Hence, the requisite functionality of a massive 208 x 208 
switch fabric 200 can be attained using one 124 x 124 
switch matrix (where 124 = 96 + 28) and one 140 x 140 
switch matrix (where 140 = 112 + 28) . In some 

20 circumstances, it may be advantageous to over-provision 
slightly and use two identical switch matrices, which in 
this case translates into a requirement for two 140 x 140 
cross-point switch matrices. Cross-point switches of 
this magnitude are available from Velio Semiconductor 

25 Corp. as part number VC 3003. Of course, other 

variations are possible, especially with respect to the 
number of input and output ports on each of the cross- 
point switches 230, 240 and the number of intra-fabric 
connections provided for loopback purposes. It should be 

30 appreciated that signal conditioning functionality (e.g., 
regeneration and re-timing) may additionally be provided 
within the switch fabric 200 or in connection therewith. 
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When signal conditioning is performed at the periphery of 
the switch fabric 200, this may be achieved through the 
use of a dedicated signal conditioning module. 

5 It should also „be mentioned at this point that if the 
signals being handled by the chassis 100 need to remain 
in an optical form throughout their journey through the 
chassis 100, MEMS (micro-electro-mechanical switch) 
devices or the like may be used in the switch fabric 200 

10 instead of the cross-point switches 230, 240. This may 
also require signal conditioning, albeit of a different 
type (e.g., re-timing and possibly multi-mode to single- 
mode conversion or vice-versa) . 

15 The connection map applied by the switch fabric 200 is 
controlled by a controller 250, which may be embodied as 
a microprocessor, FPGA, EEPROM, etc. The format of the 
connection map output by the controller 250 will, of 
course, depend on the internal structure of the switch 

20 fabric 200. In the case of the illustrated embodiment, 
the controller 250 would be responsible for providing two 
140 x 140 connection maps, one to each of the cross-point 
switches 230, 240. By changing the content of the 
connection maps, the controller 250 can change the mutual 

25 interconnection of the processor cards 130 within the 
chassis 100 and also the interconnection defined between 
the processor cards 130 in the chassis 100 and the 
external world relative to the chassis, which includes 
other chassis in a multi-chassis configuration. 

30 

The controller 250 may be located in the chassis 100 
itself,, either on a separate controller card or on the 
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interconnection module 140. The controller 250 may be 
accessed through the backplane via a dedicated external 
communication channel or it may be accessed through one 
of the line cards 110 via one of the paths P 6 4"P95- The 
5 controller 250 may be responsive to instructions 
transmitted via. a modem or other interface device (e.g., 
a communications adapter) connected over a transmission 
medium such as a tangible medium (e.g., optical or analog 
communications lines) or a medium implemented using 
10 wireless techniques (e.g., microwave, infrared or other 
transmission schemes) . 

As previously mentioned, a router may be built from two 
or more identical chassis 100 of the type illustrated in 

15 Fig. 1. The chassis are interconnected to one another 
via their interconnect modules' optical ports in 
accordance with an inter-chassis topology. Each full- 
duplex optical path travelling between optical ports may 
be carried on two separate optical fibers or on separate 

20 wavelengths of the same fiber or in any other suitable 
way known to those of ordinary skill in the art. Many 
inter-chassis topologies are within the scope of the 
present invention. In some topologies, it is desired to 
mesh all the chassis using the available optical ports. 

25 In other topologies, multiple chassis may be connected in 
a ring- or star- like configuration. In still other 
topologies, the chosen interconnect strategy will seek to 
keep the maximum number of "hops" between chassis to 
below a given upper bound. 

30 

In either case, the router so created can be upgraded by 
simply adding one or more supplementary chassis to the 
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existing group of chassis, adding new interconnections 
between previously idle optical ports and re-programming 
the interconnection modules 140 in all chassis. It is 
noted that the interconnection modules in the 
5 supplementary chassis may be pre-programmed prior to 
their interconnection to the other chassis, or they may 
be programmed once the connections have been established. 
Also of note is the fact that the hardware connections 
within each chassis and between the chassis remain fixed, 

10 and are merely appended to as the router is scaled; 
rather, it is the software that adapts to the growing 
size of the router. Hence, the router can be scaled 
without ever having to replace any hardware (thus 
minimizing the cost) and without having to disconnect any 

15 physical connections between chassis or within any of the 
chassis (thus minimizing the down time) . Thus, the use 
of a programmable interconnect module 140 within each 
chassis greatly simplifies scaling. 

20 The connection map that is provided to the switch fabric 
200 of an interconnection module 140 of a given chassis 
in the router will be a function of the chosen inter- 
chassis topology and the position of the given chassis 
within that topology- Care must therefore be taken to 

25 properly program each switch fabric 200 so as to allow 
the desired connectivity to take place. Figures 3A, 4A 
and 5A provide examples of an inter-chassis topology, for 
a router offering a progressively larger capacity. The 
accompanying Figures 3B, 4B and 5B provide examples of 

30 suitable intra-chassis interconnection possibilities that 
enable the router to achieve the requisite functionality 
at the corresponding stage of growth. 
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Accordingly, Fig. 3A shows a router 300 comprising only- 
two interconnected chassis 310, 320. For ease of 
reference, each of the chassis 310, 320 is identical to 

5 the chassis described previously with reference to Figs. 
1 and 2. Therefore, each of the chassis 310, 320 
includes four optical ports; specifically, chassis 310 
includes optical ports A 3 i 0 , B 310 , C 3 io, D 3 io and chassis 320 
includes optical ports A 320 , B 320 , C 320 , D 32 o • The inter- 

10 chassis topology is defined by optical port A 3 io of 
chassis 310 being connected to optical port A 32 o of 
chassis 320 by an optical fiber bundle 399 A . In this 
case, optical fiber bundle 399 A establishes 32 full-duplex 
optical paths between optical ports A 3i0 and A 32 o, which 

15 can be carried as 64 unidirectional paths on 64 optical 
fibers or multiplexed in any suitable way. 

Fig. 3B provides a detailed view of the interconnection 
pattern established within each of the chassis 310, 320. 

20 It has been assumed that all paths are full-duplex paths, 
although each such full-duplex path may be implemented by 
a multiplicity of physical links. For ease of 

illustration, the iine cards have, been omitted. Also for 
ease of illustration, the input and output interfaces IN, 

25 OUT of the switch fabric in the interconnection module 
140 are not specifically shown but they can be 
unambiguously determined from the correspondence 
established above with electrical paths P0-P95 (in the 
case of IN 0 -IN 95 and OUT 0 -OUT 95 ) , optical paths A 0 -A 3 i (in 

30 the case of IN 96 -IN 127 and OUT 96 -OUTi 2 7) , optical paths B 0 -B 23 
(in the case of IN 128 -INi 5 i and OUTi 28 -OUTi 5 i) , optical paths 
C 0 -C 23 (in the case of IN152-IN175 and OUT152-OUT175) and 
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optical paths D 0 -D 3 i (in the case of INn 6 -IN 2 o7 and OUT 176 - 
OUT207) • 

Turning first to chassis 310 in Fig. 3B, the 
interconnection module 140 is programmed to establish the 
following connection groups: 

(i) paths P 6 4-P7i to paths P0-P7 (loopback) 

(ii) paths P72-P79 to optical paths A 0 -A 7 of optical port 
A310 

(iii) paths P 8 o-Ps7 to paths P 8 -Pis (loopback) 

(iv) paths Pes-Pes to optical paths A 8 -A 15 of optical port 
A310 

(v) paths P16-P31 to optical paths Ai 6 -A 3 i of optical port 
A310 

With continued reference to Fig. 3B, the interconnection 
module 140 of chassis 320 is programmed to establish the 
following connection groups: 

(vi) paths P 6 4-P7i to paths P 0 -P? (loopback) 

(vii) paths P 72 -P 7 g to optical paths Ai 6 -A 2 3 of optical 
port A 3 2o 

(viii) paths P 8 o-Ps7 to paths P 8 -Pis (loopback) 

(ix) paths P88-P95 to optical paths A 24 -A 3 i of optical port 
A320 

(x) paths P16-P31 to optical paths A 0 -Ai 5 of optical port 
A 32 o 

With additional reference to Fig. 4A, the capacity of the 
router 300 of Fig. 3A and 3B can be augmented by adding a 
third chassis 330 to the existing set of chassis 310, 
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320, resulting in the creation of a router 400. Chassis 
330 has a plurality of optical ports denoted A 330 , B 330 , 
C330, D330. A fiber bundle 399 B is connected between 
optical ports B 3 i 0 and B 330 on chassis 310 and 330, 
5 respectively, while a fiber bundle 399 c is connected 
between optical ports C 32 o and C 330 on chassis 320 and 330, 
respectively. It is noted that fiber bundle 399 A , which 
joins optical ports A 3i0 and A 32 o on chassis 310 and 320, 
respectively, remains intact and that none of the 
10 hardware within any chassis needs to be replaced, 
disconnected to re-connected. 

Fig. 4B provides a more detailed view of one possible 
intra-chassis interconnection pattern which can be 

15 established by the interconnection module 140 of each 
chassis 310, 320, 330 and which allows the router 400 to 
function as a scaled version of the router 300. It is 
recalled that the actual connections (i.e., from 
electrical interface to electrical interface) of the 

20 switch fabric within each interconnection module 140 can 
be derived from the illustrated interconnection pattern 
and from the correspondence between the interfaces and 
the paths, which was described earlier with reference to 
Figs. 2 and 3B. Specifically, the following connection 

25 groups have been established within chassis 310: 

(i) paths P64-P71 to optical paths A 0 -A 7 of optical port 
A 310 

(ii) paths P72-P75 to paths P 0 -P 3 (loopback) 

30 (iii) paths P 76 -P 7 9 to paths P 36 -P 3 9 (loopback) 

(iv) paths P 80 -P87 to optical paths A 16 -A 23 of optical port 
A310 
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(v) paths Pee-Pgi to paths P32-P35 (loopback) 

(vi) paths P92-P95 to paths P4-P7 (loopback) 

(vii) paths P 8 -Pi5 to optical paths A 8 -Ai 5 of optical 
port A310 

5 (viii) paths P16-P23 to optical paths B 0 -B 7 of optical 
port B310 

(ix) .paths P40-P47 to optical paths A24-A31 of optical port 

A310 

(x) paths P48-P55 to optical paths B 8 -Bi 5 of optical port 
10 B310 

With continued reference to Fig. 4B, the interconnection 
module 140 of chassis 320 is programmed to establish the 
following connection groups: 

15 

(xi) paths P 64 -P 6 7 to optical paths A 8 -An of optical port 
A 32 o 

(xii) paths P 68 -P7i to optical paths A24-A27 of optical 
port A320 

20 (xiii) paths P72-P75 to paths P0-P3 (loopback) 

(xiv) paths P76-P79 to paths P36-P39 (loopback) 

(xv) paths P 8 o-Ps3 to optical paths Ai 2 -A 15 of optical port 
A320 

(xvi) paths P 8 4-Ps7 to optical paths A 28 -A 31 of optical 
25 port A320 

(xvii) paths P 88 -P 9 i to paths P32-P35 (loopback) 

(xviii) paths P92-P95 to paths P4-P7 (loopback) 

(xix) paths P 8 -Pi5 to optical paths A 0 -A 7 of optical 
port A320 

30 (xx) paths P24-P31 to optical paths C 0 -C 7 of optical port 
C320 
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(xxi) paths P40-P47 to optical paths Ai 6 -A 2 3 of optical 
port A320 

(xxii) paths P48-P55 to optical paths C 8 -Ci 5 of optical 
port C320 

With continued reference to Fig. 4B, the interconnection 
module 140 of chassis 330 is programmed to establish the 
following connection groups: 

10 (xxiii) paths P 6 4-Pe7 to optical paths B 0 -B 3 of optical 

port B 330 

(xxiv) paths P68-P71 to optical paths B 8 -B u of optical 
port B330 

(xxv) paths P72-P75 to optical paths C 0 -C 3 of optical 
15 port C330 

(xxvi) paths P76-P79 to optical paths C B -C U of optical 

port C 330 

(xxvii) paths P 8 o-Ps3 to optical paths B4-B7 of optical 
port B 330 

20 (xxviii) paths P 8 4-Ps7 to optical paths B 12 -Bi 5 of optical 
port B 330 

(xxix) paths P88-P91 to optical paths C4-C7 of optical 
port C330 

(xxx) paths P92-P95 to optical paths C12-C15 of optical 
25 port C330 

Thus, it has been shown that the capacity of a router 
designed according to an embodiment of the present 
invention can be increased by (1) adding a new chassis; 
30 (2) connecting one or more additional fiber bundles 
between the optical ports of the chassis; and (3) re- 
programming the interconnection module 140 in each 
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chassis via the controller 250. No existing connection 
within any chassis or between any pair of chassis needs 
to be physically dismantled or re-established, resulting 
in a truly scalable solution to the problem of increasing 
5 router capacity. It is also noted that the 

interconnection module in new chassis may be pre- 
programmed prior to its interconnection to the existing 
chassis, or it may be programmed once the connections to 
the existing chassis have been established. 

10 

With additional reference now to Fig. 5A, it is shown how 
the capacity of the router 400 of Figs. 4A and 4B can be 
further augmented without requiring the disconnection or 
re-connection of equipment, and without requiring 

15 existing equipment to be replaced. Specifically, the 
addition of three more chassis 340, 350, 360 results in 
the creation of a .router 500. Chassis 340 has a 
plurality of ports denoted A340, B340, C340, D340, chassis 
350 has a plurality of ports denoted A 350 , B 350 , C350, D 35 o 

20 and chassis 360 has a plurality of ports denoted A 3 6o, 

B360f C 3 60? D360- 

A fiber bundle 399* A is connected between optical ports 
A330 and A340 on chassis 330 and 340, respectively. A 

25 fiber bundle 399*b is connected between optical ports B320 
and B340 on chassis 320 and 340, respectively. A fiber 
bundle 399* c is connected between optical ports C310 and 
C340 on chassis 310 and 340, respectively. A fiber bundle 
399 D is connected between optical ports D 3 i 0 and D 350 on 

30 chassis 310 and 350, respectively. Finally, a fiber 
bundle 399* D is connected between optical ports D 32 o and 
D 36 o on chassis 320 and 360, respectively. It is noted 
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that fiber bundles 399 A , 399 B and 399 c remain intact and 
that none of the hardware within any given chassis needs 
to be replaced, disconnected to re-connected. 

5 Fig. 5B provides a more detailed view of suitable 
connections that can be established by the 
interconnection module within each of the chassis 310, 
320, 330, 340, 350 and 360, thus demonstrating 
scalability of the three-chassis router 400 of Fig. 4A to 

10 the six-chassis router 500 of Fig. 5A. Specifically, the 
following connection groups are established for chassis 
310: 

(i) paths P 64 -P 6 7 to optical paths A 0 -A 3 of optical port 
15 A 3 io 

(ii) paths P68-P71 to optical paths B 0 -B 3 of optical port 
B310 

(iii) paths P72-P75 to optical paths C 0 -C 3 of optical 
port C 310 

20 (iv) paths P76-P77 to paths P 0 -Pi (loopback) 

(v) paths P78-P79 to paths P34-P35 (loopback) 

(vi) paths P 8 o-P83 to optical paths A9-A11 of optical port 
A310 

(vii) paths P 8 4-P87 to optical paths B 8 -Bn of optical 
25 port B310 

(viii) paths P88-P91 to optical paths C 8 -C n of optical 
port C 3 io 

(ix) paths P92-P93 to paths P32-P33 (loopback) 

(x) paths P94-P95 to paths P2-P3 (loopback) 

30 (xi) paths P4-P7 to optical paths A4-A7 of optical port A310 
(xii) paths P 8 -Pn to optical paths B 4 -B 7 of optical 

port B310 
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(xiii) paths P12-P15 to optical paths A12-A15 of optical 
port A310 

(xiv) paths Pi6 _ Pi9 to optical paths D 0 -D 4 of optical 
port D310 

(xv) paths P20-P23 to optical paths C4-C7 of optical port 
C310 

(xvi) paths P36-P39 to optical paths A12-A15 of optical 
port A310 

(xvii) paths P40-P43 to optical paths B12-B15 of optical 
port B 310 

(xviii) paths P44-P47 to optical paths A28-A31 of optical 
port A310 

(xix) paths P48-P51 to optical paths D4-D7 of optical 
port D 310 

(xx) paths P52-P55 to optical paths Ci 2 -Ci 5 of optical port 
C310 

(xxi) optical paths Ai 6 -A 2 3 to optical paths D 8 -D 15 
(loopback) 

(xxii) optical paths Bi 6 -B 2 3 to optical paths D 16 -D 2 3 
( loopback) 

(xxiii) optical paths Ci 6 -C 2 3 to optical paths D 24 -D 3 i 
(loopback) 

With continued reference to Fig. 5B, the interconnection 
module 140 of chassis 320 is programmed to establish the 
following connection groups: 

(xxiv) paths P 6 4-Pe5 to optical paths A4-A5 of optical 
port A320 

(xxv) paths P 6 6-P67 to optical paths A12-A13 of optical 
port A320 
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(xxvi) paths P 68 -P7i to optical paths B 0 -B 3 of optical 
port B 32 o 

(xxvii) paths P72-P75 to optical paths C 0 -C 3 of optical 
port C320 

5 (xxviii) paths P76-P77 to paths P 0 -Pi (loopback) 

(xxix) paths P78-P79 to paths P34-P35 (loopback) 

(xxx) paths Pso-Psi to optical paths A 6 -A 7 of optical 
port A320 

(xxxi) paths P 82 -P 8 3 to optical paths A 14 -Ai 5 of optical 
10 . port A320 

(xxxii) paths P 84 -P 8 7 to optical paths B 8 -Bn of optical 
port B320 

(xxxiii) paths P 88 -P9i to optical paths C 8 -Cn of optical 
port C320 

15 (xxxiv) paths P92-P93 to paths P32-P33 (loopback) 

(xxxv) paths P94-P95 to paths P2-P3 (loopback) 

(xxxvi) paths P4-P5 to optical paths A 0 -Ai of optical 
port A320 

(xxxvii) paths P6-P7 to optical paths A 8 -A 9 of optical 
20 port A320 

(xxxviii) paths P 8 -Pn to optical paths B 4 -B 7 of optical 
port B320 

(xxxix) paths P12-P15 to optical paths Ai 6 -Ai 9 of optical 
port A 32 o 

25 (xl) paths P16-P19 to optical paths D 0 -D 3 of optical port 
D320 

(xli) paths P20-P23 to optical paths C4-C7 of optical 
port C 32 o 

(xlii) paths P36-P37 to optical paths A 2 -A 3 of optical 
30 port A320 

(xliii) paths P38-P39 to optical paths Ai 0 -An of optical 
port A 320 
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(xliv) paths P40-P43 to optical paths B 12 -Bi 5 of optical 
port B320 

(xlv) paths P44-P47 to optical paths A20-A23 of optical 

port A320 

5 (xlvi) paths P48-P51 to optical paths D4-D7 of optical 
port D320 

(xlvii) paths P52-P55 to optical paths C12-C15 of optical 
port C320 

(xlviii) optical paths A24-A31 to optical paths D 8 -D 15 
10 (loopback) 

(xlix) optical paths Bi 6 -B 2 3 to optical paths . D 16 -D 2 3 

(loopback) 

(1) optical paths Ci 6 -C 2 3 to optical paths D24-D31 
(loopback) 



15 



20 



The connections for the other chassis (namely, chassis 
330, 340, 350 and 360) can similarly be obtained from 
inspection of Fig. 5B by a person of ordinary skill in 
the art. 



From the above, it is again seen how the capacity of a 
router designed according to an embodiment of the present 
invention can be increased by simply (1) adding a set of 
additional chassis; (2) connecting additional fiber 

25 bundles betweenthe optical ports of the existing and 
additional chassis; and (3) re-programming the 
interconnection module 140 in each chassis via the 
controller 250. No existing connection within any 

chassis or between any pair of chassis needs to be 

30 physically dismantled or re-established, resulting in a 
truly scalable solution to the problem of increasing 
router capacity. 



Page 28 




86177-23 



It also should be noted that for the router 500 of Figs. 
5A and 5B, there are six chassis but only four optical 
ports per chassis and hence there are more chassis in the 

5 router 500 than there are optical ports per chassis. 
When this occurs, it may be useful to divide the router 
into "clusters" of chassis where, in this. case, there are 
up to four chassis per cluster. For each chassis, three 
optical ports per chassis are reserved for connections 

10 with other chassis within the same cluster, while the 
remaining (fourth) optical port is used to connect to a 
chassis in another cluster. In general, for an N-port 
chassis, where M-l of N ports are reserved for intra- 
cluster connections, the total number of M-chassis 

15 clusters that can be accommodated is (N- (M-l ) ) +1 = N-M+2 . 
In the case where N=M (as in Fig. 5A) , the maximum number 
of clusters is two, with M (or N, since N is equal to M) 
chassis in each cluster. 

20 It should be appreciated that the number of chassis 
required to achieve a certain switching capacity may 
exceed the maximum number of chassis that can be 
accommodated by clustering. Nonetheless, it is still 
possible to build a scalable router in accordance with an 

25 embodiment of the present invention. In such a case, an 
additional component needs to be introduced, namely a 
"chassis interconnection module" (CIM) . 

By way of example, Fig. 6 shows a router 600 composed of 
30 three clusters 610A, 610B and 610C. Within cluster 610A 
are contained five chassis 640 and an optional switching 
element 660, all of which connected to a common CIM 630 
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via fiber optic bundles (or DWDM fibers) 650. The ClM 
630 for a given cluster may have a larger number of ports 
than any one chassis and provides programmable optical 
interconnections . (e.g., using MEMS devices). The 
5 illustrated embodiment shows that none of the chassis 
within a given cluster is connected directly to another 
chassis within that cluster Such chassis-to-chassis 
connections are provided by " loopback" functionality of 
the CIM 630, which allows ' connecting two chassis together 

10 within the same cluster and connecting two other CIMs via 
the CIM 630. Nevertheless, it is within the scope of 
the present invention to provide chassis-to-chassis 
connections within the same cluster, which may be 
advantageous when implementing a hypercube-based 

15 topology. 

The clusters themselves are interconnected via the CIM in 
each cluster. An ultra-dense optical link may 

interconnect the CIMs in the various clusters. For 

20 instance, in the illustrated embodiment, link 620 
connects the CIMs in clusters 610A and 610B, while link- 
622 connects the CIMs in clusters 610A and 610C. A link 
624 emanates from CIM 630 and is unused for the time 
being but is available for future use, in case another 

25 cluster is added to the router 600. The ultra-dense 
links 620, 622, 624 may be DWDM optical fibers or they 
may comprise bundles of single-carrier or coarsely 
multiplexed optical fibers. 

30 In this way, different numbers of clusters can be 
interconnected to form a router. Scalability of the 
router is achieved by adding one or more clusters (with 
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respective CIMs) , adding selected connections between 
CIMs and re-programming the CIMs. No intra-cluster 
connections need to be disabled or re-connected. Careful 
observation of this architecture reveals similarity with 
5 the architecture of Fig. 1, where the CIM of Fig. 6 plays 
the role of the interconnection module 140 of Fig. 1. 
Thus, one skilled in the art who has understood how the 
interconnection module can be reprogrammed to provide the 
required connectivity will also understand how the same 
10 can be achieved by re-programming the CIM 630 . 

It should be understood, of course, that the above 
described functionality of the chassis and routers can 
apply to multiple individual streams of data flow, 

15 sometimes referred to as "data planes". Thus, although 
the above description has been formulated in . terms of a 
single data plane, it should be understood that the 
present invention is equally applicable to the 
transmission of information across multiple independent 

20 planes which draw upon the resources of a common set of 
chassis. The line cards and processing cards in each 
chassis may either be dedicated to . a particular data 
plane or may be shared amongst two or more date planes . 

25 Also, the term "card" is meant to be interpreted broadly 
so as to cover not only a printed circuit board that has 
" connectors disposed primarily along an edge thereof, but 
also other modules that contain circuitry, software 
and/or control logic capable of providing the requisite 

30 functionality. A large-area wafer comprising all the 
requisite internal circuitry of one or more "cards" is 
also envisaged. 
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While specific embodiments of the invention have been 
described and illustrated, those skilled in the art will 
appreciate that further modifications and variations may 
be made without departing from the scope of the invention 
as defined in the claims appended hereto. 
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